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Within the population of uoiceband telephone channels, few channel 
characteristics are as pervasive in their impairment of high-speed data 
communication as nonlinear distortion, which cannot be removed or 
equalized in the receiver as easily as can linear distortion. The purpose 
of this paper is to report on an investigation of a QAM receiver incor- 
porating adaptive equalization of nonlinearities as well as adaptive 
decision feedback equalization and data-aided carrier recovery for 
mitigation of linear distortion and phase jitter, respectively. Non- 
linearities are equalized by adding to the received in-phase and 
quadrature signals a weighted sum of nonlinear functionals of the re- 
ceived signal and of modulated previous receiver decisions. The choice 
of nonlinear terms in the sum is based on a channel model incorporating 
quadratic and cubic nonlinearities as well as linear dispersive elements. 
The adjustment of the weighting, or tap, coefficients for the various 
terms is based on a gradient algorithm, as is the adjustment of the 
linear tap coefficients and the carrier phase reference. The feasibility 
of nonlinearity equalization on real voiceband channels was confirmed 
in a test in which recorded 9600-bps QAM signals, received from a 
worse-than-average set of 17 voiceband telephone channels, were 
processed by a computer-simulated version of the proposed receiver 
(termed the NL receiver). The observed error rates for all channels were 
lower, in some cases by several orders of magnitude, than those achieved 
by computer-simulated versions of the linear receiver and of a decision 
feedback equalization receiver (termed the DFE receiver). 

I. INTRODUCTION 

The prevalence of nonlinearities and their distorting effect on high- 
speed data transmission over voiceband telephone channels has long 
been recognized. 1 The effect of nonlinear distortion on linearly modu- 
lated data signals is to introduce nonlinear intersymbol interference and 
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reduce the margin against noise. For data rates above 4800 bps, nonlinear 
distortion is the dominant impairment on many voiceband telephone 
channels. Experimental studies have measured nonlinear distortion and 
related the observed error rates for specific modulation formats to this 
and other measured impairments. 2 ' 3 Estimation of performance for data 
transmission in the presence of nonlinearities can be carried out 4 but 
gives little insight into the problem of receiver optimization, except for 
certain simple nonlinear channel models. 5 

Recognizing that nonlinearities in transmission channels generally 
coexist with linear elements such as filters, one is led to consider a general 
nonlinear receiver structure, based on a Volterra or Wiener kernel 
characterization 6 of a general nonlinear system such as that proposed 
in Refs. 5, 7, and 8, the latter in connection with adaptive echo cancel- 
lation. In the present work, we extend this approach by generalizing the 
structure of a passband decision feedback equalizer, previously studied 
in connection with linear channel distortion, 9 to process nonlinear as well 
as linear functionals of the incoming signal and prior decisions.* 

The new receiver structure is based on a model of a passband channel 
with quadratic and cubic nonlinearities, as well as linear filters. We re- 
port on the simulation of the new receiver and on comparisons of its 
performance with two other previously simulated 9600-bps QAM re- 
ceivers on a worse-than-average set of voiceband telephone channels. 
The new receiver is referred to as the NL receiver. The other two re- 
ceivers, designated LE (linear equalization) and DFE (decision feedback 
equalization), are not designed to compensate for channel nonlinearities. 
Their performance is compared over the same set of voiceband telephone 
channels in Ref. 9. The simulated LE receiver is described in Ref. 10. 



II. SUMMARY OF THE MAJOR RESULTS 

The relative performances of the three simulated receivers on the same 
set of recorded, received, 9600-bps data signals are briefly summarized 
as follows: On every channel, the NL receiver yielded a lower error 
probability than the other two receivers. For 13 out of the 17 channels, 
the improvement in error rate was equal to or better than about an order 
of magnitude. Another gauge of the degree of improvement offered by 
the NL receiver is the fact that it increased the number of channels 
yielding a better-than-10 -4 error rate from 8 to 15. On one channel, 
whose major impairment was second harmonic distortion, the NL re- 



* Figure 3a summarizes the structure of the nonlinearity-equalizing receiver. 
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ceiver's error rate bested that of the DFE and LE receiver by over four 
orders of magnitude. Figure 5 is a bar graph summarizing the error rate 
comparisons. 

The apparent attractiveness of the NL receiver structure is, however, 
tempered by its greater complexity. A large number of nonlinear tap 
coefficients is necessary to account and compensate for the dispersive 
nonlinear effects typically encountered on voiceband channels. In the 
simulations summarized above, the LE and DFE receivers each had 32 
complex tap coefficients, but the NL receiver was, roughly speaking, 
comparable in complexity to an LE receiver with 134 complex tap coef- 
ficients. Reducing the number of coefficients in the NL receiver lowered 
its performance margin over the other receivers. Furthermore, the best 
allocation of a fixed number of tap coefficients varied from one channel 
to another. These points are explored more fully in later sections. 

In spite of the greater complexity of the NL receiver structure, the 
performance comparison of the three receivers does indicate the im- 
portance of alleviating nonlinear distortion for high-speed data trans- 
mission. 



III. THE CHANNEL MODEL 

Obviously, the effect of channel nonlinearities on a passband QAM data 
signal must be understood before a compensating receiver structure can 
be suggested. A general representation of a bandlimited QAM signal is 
as the real part of a complex waveform: 



x(t) 



= Re ["eJarf'' L A{n)F(t - nT)\ (1) 



where; = v/^1 , f c is the carrier frequency, A(n) is a quantized complex 
number representing the information symbol in the nth symbol interval 
(for example, in the case of four-level QAM, the real and imaginary parts 
of A{n) assume one of the four possible values ±1, ±3), T is the reciprocal 
of the baud, and F(t) is a complex pulse waveform. 

In the case of QAM signals, extraction of the information symbols 
represented by the complex number A (n) requires two receiver outputs, 
which are derived by appropriate operations on both the received 
passband signal and on its quadrature version, or Hilbert transform. A 
phase-splitting filter is used to obtain both in-phase and quadrature 
versions of a voiceband data signal. 

The complex waveform 

X(t) = e>**« £ A(n)F(t - nT) (2) 
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is assumed analytic; 11 that is, its spectrum is twice the Fourier transform 
of x(t) for positive frequencies and is zero elsewhere. Furthermore, we 
assume the spectrum is limited on the high side to frequency 2/ c . Note 
that the Fourier transform 3(f) of the complex pulse F(t) is not neces- 
sarily symmetric about / = 0, but it is assumed to be strictly band-limited 
to —f c < f < f c . The Nyquist frequency is 1/2T Hz. Figure 1 shows a 
sketch of 3(f) and of 3(f — f c ), which is the Fourier transform of 

eJ*/«*F(t). 

The notion of analytic signals is a notational convenience. The Hilbert 
transform, or quadrature version of a signal u(t), is a linear functional 
ofu(t): 



u(t) = — I dr. 

■K J — t — T 



It can be shown that there is a unique analytic signal whose real part is 
u(t), and that u(t) is then just the imaginary part of the analytic signal. 
Conversely, any analytic signal comprises some real signal plus j times 
its Hilbert transform. Since QAM systems operate on both in-phase and 
quadrature versions of signals, they are most conveniently represented 
by means of analytic signals. 

The nonlinear receiver structure will be based on the simple nonlinear 
channel model shown in Fig. 2, using the notation of analytic signals. 
Filters 1, 2, and 3 are passband with the same bandwidth as the trans- 
mitted data signal. The filters may include the receiver's input filter as 
well as the linear response of the channel. The quadratic and cubic 
memoryless nonlinearities with attenuated outputs account for second 





f c 2f c f 

Fig. 1— Fourier transforms of \F(f) \ and \F(f-c)\. 



2592 THE BELL SYSTEM TECHNICAL JOURNAL, SEPTEMBER 1978 





FILTER NO. 1 










]r-\, 

2f c 


x(I) 


I 


r^~ 




INPUT 
DATA 




' 


L 


- 

s 




SIGNAL 












FILTER NO. 2 












SQUARE 


x(t) 2 




k - ^ 




^ 








2f c 














FILTER NO. 3 










CUBE 


x(t) 3 


l/-\, 














2f c 











Fig. 2 — Model of a nonlinear channel. 

and third harmonic distortion, respectively. Additional impairments 
not shown in Fig. 2 are phase jitter, which implies multiplication of the 
complex received signal by e^ (t) , and additive noise. 

The result of passing the transmitted waveform through the linear 
portion of the channel (filter 1) is an analytic waveform in the form of 
eq. (2). A passband linear 10,12 equalizer (le) can be used to minimize the 
mean squared error between its output, sampled at times nT, and a 
reference A{n)eJ (2 *f cnT +' e(n) \ which is the complex information symbol 
modulated to passband with a receiver phase reference 6{n). In a linear 
receiver, the passband equalizer output is demodulated [multiplied by 
e -j(2rf c nT+kn))] ^j t nen quantized to yield a decision A(n). A passband 
equalizer configuration which is theoretically more effective in com- 
batting linear intersymbol interference is the passband DFE, described 
in Ref. 9. 

To motivate a receiver structure which is appropriate for nonlinear 
distortion as well as linear distortion, we must consider the analytic 
signals emanating from the quadratic and cubic path elements of Fig. 
2. 

It is shown in the appendix that the analytic signal output from the 
model of Fig. 2 is of the form 

R(t) - U (t) + eft'teUiid) + e-i**f*Uu(t) 

+ ei^fc'Uzit) + eJe^'Uait), (3a) 



where 



U (t)= E A( ni )A(n 2 )*Go(t - mT,t - n 2 T) 

ni,n2 



(3b) 



U u (t) = ZMn)F(t-nT) 



+ £ A(n 1 )A(n 2 )A(n 3 )*G ll (t-n l T,t-n 2 T,t-n 3 T) (3c) 

"1.12,13 
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Unit) = £ A(n 1 )*A(n 2 )*A(n 3 )G 12 (t - n x T,t - n 2 T,t - n 3 T) 

(3d) 
U 2 (t) = £ A(n 1 )A(n 2 )G 2 (t - n x T,t - n 2 T) (3e) 

n\,n 2 

U 3 (t) = £ A( ni )A(n 2 )A(n 3 )G 3 (t - n t T,t - n 2 T,t - n 3 T), 

"1.12,13 

(3f) 

where asterisks denote complex conjugates. 

The various U terms are seen to be linear combinations of products 
of complex information symbols A(n), A(n\)A(n 2 ), A(ni)A(n 2 )A(n 3 )*, 
etc. Each modulates a harmonic of the carrier wave. The term ei 2ir f ct - 
Uu(t) includes the linear response of the channel to the data signal and 
also a component resulting from cubic distortion. The terms Uo(t) and 
U 2 (t) result from the quadratic nonlinearity and the terms Unit) and 
U 3 (t) result from the cubic nonlinearity. Additional terms would, of 
course, result from the assumption of additional nonlinear elements in 
the model of Fig. 2. The generalization of expression (3) to an infinite 
power series would be a complex passband version of a Volterra expan- 
sion. 

IV. THE NONLINEAR RECEIVER STRUCTURE 

The receiver structure to be studied here includes the passband QAM 
decision feedback equalizer discussed in Ref. 9, plus nonlinear processing 
suggested by the set of eqs. (3). Let Y(n) be the receiver's complex output 
at time t = nT. This output is quantized to form the decision A(n), which 
equals the original transmitted symbol A(n) if no error occurred. Let 
the demodulator's phase reference at time nT be 0(rc). Let [W^^-n 
and {B^jfeii be the complex linear forward and feedback tap coefficients 
respectively, and let \R(n)\ be the complex receiver input, sampled at 
times nT. Then 

N M 

Y(n) = e-J^fcnT+Hn)) £ w p* R ( n _ k) _ £ B j,D* A(n-k) 

k=-N k=l 

+ Y NL (n)e-J< 2 *fc nT+ ' e < n », (4a) 

where YnlM consists of nonlinear functions of \R(k)\ and \A(k)\k< n - 
The linear part of eq. (4a) implies a demodulated linear combination 
of 2N + 1 receiver input samples minus a linear combination of M pre- 
vious decisions. 

The nonlinear term YmlM is heuristically suggested by expression 
(3) in the following way: (i) Assume that at time nT the previous receiver 
decisions A(k) = A(k){k <n) and that they are available to form the 
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nonlinear feedback terms, (ii) In any terms of expression (3) involving 
decisions A(k) not yet made at time n(k > n), replace A (k) e J 2 *f< kT+h{k) 
hyR(k) to form the forward nonlinear terms. The resulting expression 
is 

Y NL (n)= £ Wio&Rin-kJRin-^)* 

k\,k2 

+ £ W[\% M R(n- fei)/?(n - k 2 )R{n - k 3 )* 

fcl.fc2.fc3 

+ L Wi l X.kaR(n-k 1 )*R(n-k 2 )*R{n-k s ) 
fc1.fc2.fc3 

+ E WJSj t2 R(n-ki)R(n-k 2 ) 

k\,k2 

k\,k 2 
>i 

_ e j(2,f c nT+Hn)) £ Bggjk, ^(n - ki)A(n - fe 2 )A(n - fe 3 )* 

fcl.fc2.fc3 

>1 

_ e -H2icf e nT-Kn)) £ Bjggjk, A(n - fei)*A(n - k 2 )*A(n - k) 

fcl.fc2.fc3 

>1 
_ e j(4*f e nT+Hn)) £ Bffi 2 A(n - k^)A{n - k 2 ) 

>i 

X A(n -kx)A(n - k 2 )A(n - fe 8 ). (4b) 

The formidable-looking expression (4b) is a linear combination of 
products of receiver inputs and their complex conjugates, minus a linear 
combination of products of previous decisions and their complex con- 
jugates, modulated by appropriate harmonics of the carrier. 

Figures 3a and 3b are block diagrams of the NL receiver. The cross- 
hatched boxes in Figure 3a show the nonlinear processing that has been 
added to the basic decision feedback equalization structure described 
in an earlier paper. 9 

V. ADAPTATION OF RECEIVER PARAMETERS 

As in the linear and decision feedback equalization receivers, the pa- 
rameters \W\, [B\ and are adjusted in an estimated gradient algorithm 
to minimize the average value of the squared error magnitude |£(n)| 2 
defined by 
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E(n)=Y(n)-A(n). (5) 

The error E(n), as in the previous receivers, is a linear function of the 
parameters [W\ and {£}; consequently, the expression for |jE(n)| 2 is 
convex in these parameters. 

In writing the updating equations for the { W(n)\ and \B(n)\ coefficients 
anp! for Q(n) in the nth symbol interval, it is convenient to use the symbol 
e (n) to denote the observed passband error after the decision A (n) has 
been made: 

e(n) = [Y(n) - A{n)]e^ 2 ^ nT+h ^\ (6a) 

thus, if A(n) = A(n), \E(n)\ 2 = |c(n)| 2 , and the expression for the gra- 
dient of | e(n) | 2 with respect to each parameter determines an adjustment 
algorithm for that parameter. The adjustment equation for 6(n) is as 
follows: 

0(n+l) = 6(n) , % . , l9 , (6b) 

where Z(n) = L W^* R(n - k) 

k = -N 

+ L W^ f k2 R(n-k 1 )R(n-k 2 )* 

+ £ W[\% M R(n - kJRin - k 2 )R(n - k 3 )* 

+ £ Wi\% M R(n - k&Rin - k 2 )*R(n - k s ) 

+ L WiZ 2 R(ri-ki)R(n-k 2 ) 

k\,k2 

+ E WjgjntoRin-kdRijn-kMifi-hd (6c) 

is the sum of all the forward terms comprising Y(n). The adjustment 
equations for the {W\ and \B\ coefficients are as follows: 

W[°; M {n + 1) = Wf! M (n) - fo(n)*R(n - kJRin - k 2 )* (6d) 

Wi 1] (n + 1) = W[ l) (n) - Mn)*R(n - k) (6e) 

Wlta (n + 1) = Wi\% M (n) - u e(n)*R(n - k x ) 

-R(n-k 2 )R{n-k 3 )* (6f) 

Wi\% M (n + 1) = W[\% M (n) - (3 12 e(n)*R(n - fe x )* 

• R(n - k 2 )*R(n - k 3 ) (6g) 

WJ$ M (n + 1) = Wfi kt (n) - (3 2 e(n)*R(n - k^Rin - k 2 ) (6h) 

W^mm (n + 1) = Wf! MM (n) - Mn)*R(n - k t ) 

• R(n - k 2 )R(n - k 3 ) (6i) 
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Bi% 2 {n + 1) - Bi% 2 (n) + 7oe(")*A(n - k x )A{n - k£*eMn) (6j) 
BP (n + 1) - Bi" (n) + 7ie(»)*A(n - fe) e i(2«/e»T+»(»)) (6k) 

Affile <»+l>-3i8U»t») 

+ 7ll 6(n)M(n - kjAin - k 2 )A(n - fe 3 )*ei(2«/c»r+d(n)) (61) 

+ 7i 2 e(n)*A(n - k x )*A{n - k 2 )*A(n - fe 3 )e-i(WcnT+»(n)) ( 6m) 
Bi?* 2 (/i + 1) = Bg* (n) + y 2 e(n)*A(n - fe x ) 

• i(/l - fe 2 ) e i(4T/cnT+8(n)) ( 6n ) 

Bfii^diH-D-Bfiu^Oi) 

+ 7 3 e(n)*A(n - fei)A(n - k 2 )A(n - k 3 )eJl**f' nT+i < n » (6o) 

The set of eqs. (4) through (6) defines the structure of the nonlinear 
QAM receiver that has been simulated. The a, /3, and 7 parameters are 
positive constants, chosen to ensure reasonably fast convergence and 
stability in the presence of noise. To enable compensation of rapidly 
varying phase jitter, the phase tracking constant a was set to the rela- 
tively large value of 0.4. The other constants chosen were: 

ft = Tl B 0.001, ft) = 02 = 7o = 72 = 0.75 X 10" 5 , 

#11 = 012 = 03 = 711 = 712 = 73 = 10~ 6 . 

A judicious choice must be made for the range of coefficient indices 
k\, k 2 , and k 3 in the nonlinear terms making up YnlM, if the total 
number of ( W\ and \B\ coefficients is to be reasonable, say on the order 
of 100. Obviously, the best choice of indices for a fixed number of taps 
depends on the channel. Trial and error (by no means exhaustive) of 
various sets of indices used in simulations on several voiceband channels 
led to the choice of terms shown in Table I. There are 73 "forward" tap 
coefficients | W\, of which 22 are linear, and 61 "feedback" tap coefficients 
\B\, of which 10 are linear. Note that the nonlinear forward tap indices 
are confined to the range — 1 < k < 1 and the nonlinear feedback tap 
indices have been confined to the range 1 < k < 3. 



VI. THE SIMULATIONS 

The nonlinear QAM receiver structure described in the previous section 
was simulated on an IBM 360 computer to process recorded 9600-bps 
QAM data signals that had been received from 17 voiceband telephone 
channels. The simulation effort was an extension of that described for 
linear and decision feedback QAM receivers in Refs. 3 and 9, respectively. 
The set of recorded QAM signals was the same, permitting the perfor- 
mance of all three receiver types to be compared under identical con- 
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ditions. The set of 17 channels could be described as "worse than aver- 
age." Every channel had at least one impairment equal to or worse than 
the 90-percent point on the nationwide toll connection survey. 2 

The transmitted QAM signals had been generated digitally, with two 
pseudorandom four-level information symbol streams in quadrature, 
each repeating after 256 symbols. Each quadrature pair of symbols 



ADAPTIVE EQUALIZATION OF NONLINEARITIES 2599 



therefore conveyed four information bits and the symbol rate was 2400 
bauds, making a total bit rate of 9600 bps. The carrier frequency f c was 
1650 Hz, and the double-sideband baseband pulse signal had 12 percent 
roll-off. 

The received signals that were recorded in digital form (12-bit samples, 
24-kHz sampling rate) were received from a variety of real and analog- 
simulated voiceband telephone channels in tandem with an actual 50- 
km, C2-conditioned, N2-carrier voiceband channel. 

As in the simulation of the linear and decision feedback receivers, the 
adaptive passband signal processors [defined by the set of eqs. (4) and 
Table I] were preceded by a pair of fixed digital filters that split the in- 
coming signal into in-phase and quadrature components. Each was 
sampled at time instants t = r + nT (n = 0,1,2,- • •)• Each simulation was 
actually of five separate receivers in parallel, with sampling epochs r = 
0, 0.2T, 0.4T, 0.6T, and 0.8T. The results reported in this paper are in 
each case for the timing epoch which yielded the best performance. As 
noted previously in Ref. 9, the decision feedback structure generally 
produced a relatively small performance spread between the best and 
the worst timing epochs. The receiver's decisions A (n) were formed by 
quantizing each equalized demodulated output, in-phase or quadrature, 
into one of the four possible levels ±1, ±3. 

Before tabulating the simulation results, we mention some qualitative 
observations. In the interest of reducing the large numbers of nonlinear 
coefficients, it would have been desirable that only a few of the observed 
coefficients be large enough to be significant for all the channels. Un- 
fortunately, this was not the case; no pattern was discernible common 
to all channels of a significant subset of coefficients; typically, the non- 
linear component Ynl(h) in the receiver's output consisted of a large 
number of small terms, rather than a small number of relatively large 
terms plus insignificant terms. 

Another qualitative observation was that the best values for the ad- 
aptation parameters for the nonlinear coefficients were so small that 
convergence of the nonlinear tap coefficients required at least 2000 
symbol intervals, much slower than the convergence rate of the linear 
coefficients. This is attributed to the high correlation among many of 
the nonlinear terms. For example, the term | A^ a 1 2 A^ 2 is positively 
correlated with the linear term A* 2 , since | A* J 2 takes only one of the 
three possible positive values 2, 10, or 18. Under such circumstances, the 
Jl matrix which describes the correlations among all the terms is ex- 
pected to have a rather large eigenvalue spread, necessitating small 
adaptation constants and slow convergence. 13 

During each run, after an initial training period of 2000 symbol in- 
tervals to allow the coefficients to converge to nearly stationary values, 
the simulated receivers switched to a decision-directed mode in which 
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their decisions A(n), right or wrong, were used in the adaptation and 
decision feedback operations. Since the true transmitted information 
stream (A(n)} was known, the performance was measured by observing 
the number of decision errors made during 7000 symbol intervals (or 
28,000 bits). The empirical probability of the sampled analog error Y(n) 
— A(n) was also measured, and if no errors were observed during a run, 
the error probability could be roughly estimated by extrapolating the 
tail of this distribution, using a computer subroutine by S. B. Wein- 
stein. 14 The tabulated error probability, p e , is the probability that a 
four-level symbol is in error; i.e., it is roughly twice the bit error rate. 
Another tabulated measure of performance was the output SNR, defined 

by 

<|A(n)| 2 > 

° UtpUtSNR= OEW 
where " ( ) " denotes the time average. 

VII. QUANTITATIVE RESULTS 

The simulation results for the NL receiver are tabulated in Table II 
along with the corresponding results taken from Ref. 9 for the LE and 
DFE receivers. For each channel, Table II lists the measured impairments 
and the error probabilities (either observed or extrapolated) for the LE, 
DFE, and NL receivers. The quantity in parentheses below each error 
probability is the output SNR in decibels. Error rates below 10~ 5 were 
extrapolated; in some cases in which the tail of the empirical probability 
distribution of the quadrature components of E{n) was markedly non- 
Gaussian, the extrapolation yielded limited accuracy. Figure 4 illustrates 
the nonlinear compensation for channel 14, which had unusually severe 
second-harmonic distortion. Figure 4 is plotted on a "probability scale;" 
i.e., a Gaussian error distribution function would plot as a straight line 
on it. The distribution function for the linear receiver has distorted tails, 
indicating the presence of residual nonlinear distortion. However, the 
curve is nearly straight for the NL receiver, indicating that nonlinear 
distortion components have been substantially removed. 

Comparison of error rates for the three receivers on all the channels 
is displayed more dramatically by the bar graph of Fig. 5. In all cases, 
the performance of the NL receiver surpassed that of the other two re- 
ceivers. (Note that measurable nonlinear distortion was observed on all 
the channels.) In most cases, the NL receiver afforded a greater im- 
provement in error rate over the DFE receiver than did the DFE receiver 
over the LE receiver. This is a very significant point. It indicates that if 
9600-bps voiceband modems are to be improved by more sophisticated 
signal processing at the receiver, it is more fruitful to attempt to over- 
come nonlinear distortion than to concentrate on more sophisticated 
receiver structures, optimal for linear channel models. 
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Fig. 4 — Comparison of distribution functions of the receiver output Y(n) for the linear 
and nonlinear receivers (data from channel 14). 

Note in Fig. 5 that, for some of the channels, the nonlinearity equali- 
zation reduced the error rate by two or three orders of magnitude. 
However, on other channels, such as 9 and 16* which had most of their 
impairments in the "severe" category, the error rate was high and the 
NL receiver afforded very little improvement. 

An interesting statistic that can be gleaned from Fig. 5 concerns the 
ability of the NL receiver to increase the number of channels which yield 
error rates below a specified maximum. For example, 15 of the 17 
channels yield an error rate of better than 10~ 4 with the NL receiver, but 
only 8 of 17 meet this error rate standard with the LE receiver. For a 
maximum error rate of 10 -5 , the number of channels is 10 with the NL 
receiver and 7 with the LE receiver. For a maximum error rate of 10 -6 , 
the numbers of channels are 9 and 3 with the NL and LE receivers, re- 
spectively. 

The price paid for the better performance of the NL receiver is, of 
course, its increased complexity, measured by the number of terms 
comprising YnlM in eq. (4) and its slower convergence. The effect of 
reducing the number of terms, and therefore the complexity, is treated 
in the next section. 



* Channel 16's impairments, produced by a line simulator, were all "worst case' 
values. 
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VIII. MODIFICATIONS OF THE NONLINEAR RECEIVER STRUCTURE 

8. 1 Reductions of the number of nonlinear tap coefficients 

(i) The tap coefficients {W^l^jl and (B^^g) were set to zero, re- 
ducing the total number of nonlinear forward and feedback taps to 37 
each. The measured output SNRs for most of the channels were slightly 
less than those for the full complement of 51 forward and 51 feedback 
taps, as illustrated in Table III. 

(ii) A different set of 100 nonlinear terms was created by eliminating 
all cross-product terms and extending the time span covered by the 
forward and feedback terms to 10 symbol intervals. Thus, the forward 
tap coefficients consisted of [Wi%], {Wgfo}, \Wi%\, \W^\ and (Wffij, 
where — 5 < k < 4, and the feedback terms consisted of {£$}, {B$^}, 
\Bi%\, \Bi% and \Bi% k ], where 1 < k < 10. Some resulting output SNRs 
are tabulated in part (Hi) following. 

(Hi) A smaller set of nonlinear taps was created by taking a subset of 
46 of the original set of 102 nonlinear taps. The resulting output SNRs 
for several channels are shown in Table IV, along with the corresponding 
set of SNRs from the original NL receiver structure with 102 nonlinear 
taps and also from the receiver with 100 nonlinear taps, described in item 
(ii), above. 

The results of items (i), (ii), and (Hi), compared with the original re- 
sults using the NL receiver with 102 nonlinear tap coefficients indicate 
that a large number of nonlinear correction terms is necessary to yield 
substantial performance improvement. Undoubtedly, still better per- 
formance would have been attained by using more than 102 nonlinear 
taps. The results of item (ii) also showed that elimination of the cross- 
product terms degraded performance, even though the remaining non- 
linear terms encompassed a longer time span. 

Table III — Output snr (dB) for nonlinear receivers 



Channel 


102 nonlinear taps 


74 nonlinear taps 


5 




29.4 




29.4 


6 




24.9 




23.8 


7 




23.7 




22.7 


8 




22.3 




22.1 


9 




18.2 




17.5 


10 




24.7 




24.5 


11 




21.7 




21.5 


12 




28.6 




27.5 


Table IV 


— Output snr (dB) for nonlinear 


receivers 




Original 








Channel 


(102 Taps) 


(ii) 100 Taps 




(ui) 46 Taps 


9 


18.2 


17.7 




17.5 


13 


27.5 


25.8 




25.7 


14 


26.8 


23.7 




23.0 


15 


28.4 


25.6 




26.5 
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(iv) The number of nonlinear taps was also reduced to 46 by elimi- 
nating all coefficients IWgJ, |BgJ, { H^hJ, and \Bi* MM \. The re- 
sulting output SNR on channel 14 was only 21.3 dB, as compared to 24.8 
dB for 102 nonlinear taps. Thus, it appears that at least the last four sets 
of coefficients (associated with second and third harmonics of the carrier 
frequency) are significant and should be retained. 

8.2 A variation In the receiver structure tested for channel 20 

The forward nonlinear tap coefficients weight various quadratic and 
cubic products of the sampled received signals. One might speculate that 
if linear distortion were removed from the received samples before their 
nonlinear processing, the nonlinear distortion remaining in the output 
might be further reduced. Accordingly, we simulated an NL receiver 
structure which was the same as that shown in Fig. 4 except that there 
are no linear feedback taps and the input to the forward nonlinear taps 
comes from the output of the linear forward taps instead of directly from 
the phase splitter. Since the adaptive linear forward taps, constituting 
the passband equilizer, are in tandem with the adaptive nonlinear taps 
in this structure, the mean squared error is not a convex function of the 
nonlinear tap coefficients, and hence the question of convergence is more 
complicated. Nevertheless, this structure was simulated on channel 20. 
The resulting output SNR was 20.0 dB compared to the 20.6 dB obtained 
from the original receiver structure. Thus, prior linear equalization did 
not appear preferable. 

IX. CONCLUSIONS 

The simulations have demonstrated that nonlinearity-equalizing QAM 
receivers can provide substantially better performance than can con- 
ventional linear or decision feedback equalization receivers over a variety 
of voiceband telephone channels. This encouraging result may stimulate 
further research aimed at finding less complicated receiver structures 
for overcoming channel nonlinearities. 

The number of nonlinear terms that can be considered for inclusion 
in the NL receiver's analog output Y(n) is potentially enormous. For 
example, the number of different terms RikJRfaWikz)* for all indices 
k\, k 2 and k 3 between — N and +N is (2N + l) 2 (iV + 1), which is much 
more than (2AT + 1), the corresponding number of linear terms [R(k)\ 
in that range of indices. The simulation results indicated that inclusion 
of a large number of nonlinear terms, including "cross-product" terms 
for which k\ ?± k 2 ^ k%, may be necessary. Reductions in the number 
of terms and a variation of the NL receiver's structure, in which adaptive 
linear processing preceded nonlinear processing, resulted in worsened 
performance. 

Perhaps the major conclusion to be drawn concerns means for im- 
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proving the reliability of high-speed data transmission over the popu- 
lation of voiceband telephone channels. The simulations reported in Ref. 
9 showed that decision feedback equalization, which is known theoret- 
ically to be superior to linear equalization in overcoming severe linear 
distortion, only moderately bettered the error rate obtained with linear 
equalization, especially on voiceband channels meeting C2 conditioning 
standards. However, the results summarized by Fig. 5 indicated that 
there is more to be gained by mitigating nonlinear distortion than in 
using more elaborate methods (beyond linear or decision feedback 
equalization) of mitigating linear distortion. 

APPENDIX 

In this appendix, we derive the form of the analytic signal that emerges 
from the summed filtered outputs of the quadratic and cubic non- 
linearities. The real and imaginary parts of this analytic signal will then 
be the in-phase and quadrature components, respectively, of the non- 
linearly distorted received QAM signal. The following theorems, proven 
in Ref. 11, will be required: 

Theorem 1: Given real waveforms u{t) and v(t), defined on — °° < t < 
0° with respective Hilbert transforms u(t) and v(t), the convolution 

w(t)= f " v( T )u(t-T)dT (7) 

has Hilbert transform 

w(t)= f" v(r)u(t - r)dr = f "* v(r)u(t - r)dr. (8) 

Thus, if v(t) is the input to a filter whose impulse response is u(t) , the 
analytic output signal is 

W(t) + jw(t) - C (u(t) + jv{ T ))u(t ~ T)d 



>(t)+jw(t) = f ~ (v(t) + jv(r))u(t - 



T)ar 



-£ 



v(r)(u(t - t) + ju(t - r))dr. (9) 



Theorem 2: The analytic signal resulting from the convolution can also 
be expressed as 



w 



(t) + jw(t) m i P ( V (t) + jv(r))(u(t - t) + ju(t - r))dr. (10) 
Now we consider an analytic signal of the form 

X(t) = eJ^fct £ A(n)F(t - nT), (11) 
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as in expression (2) of the text. The squaring and cubing elements in Fig. 
2 operate on x(t), the real part of X(t). The response of the squaring 
element to Re(X(£)) can be written 

x(t) 2 = ^Re[e> 4 ^ £ A(n 1 )A(n 2 )F(t - ni T)F(t - n 2 T)] 

+ | F I A(in)A(n 2 )*F(fc - mT)F(t - n 2 T)*l. (12) 

£ Lni,n2 J 

Of the complex expressions in square brackets in (12), the first is 
complex and analytic, since it is the square of an analytic signal (its 
spectrum is nonzero only for positive frequencies). Thus, from Theorem 
2, the analytic signal that results from passing the first part of expression 
(12) through a passband filter 2 is of the form 

U 2 (t) = eM* L £ A(n 1 )A(n 2 )G 2 (t - n x T t t - n 2 T), (13) 

n\ n.2 

where G 2 (t — n\T,t — n 2 T)ei 4 * fct is a complex analytic waveform, whose 
spectrum has been limited by filter 2 to < f < 2/ c . The second term in 
(12) is baseband, real, and not analytic.* However, from Theorem 1, the 
analytic signal resulting from passing the second term through filter 2 
has the form 

U (t)= £ A(ni)A(n 2 )*'G (t - ni T,t - n 2 T), (14) 

where Go(t — niT,t — n 2 T) is an analytic waveform, whose spectrum is 
confined to < / < 2/ c . 

The cubic nonlinear terms are handled similarly. The cube of the input 
signal Re(X(t)) can be written 

x(t)*-^Re\eJ**fct £ AinJAinMinzWit - mT) 
4 L 11.n2.n3 

. F(t - n 2 T)F(t - n 3 T)] 

+ |««W £ A(m)A(n2)Ains)*F(t - mT)F(t - n 2 T) 

" n1.n2.n3 

•F(t-n d T)* + K-&*fct £ A(m)*A(n 2 )*A(n 3 ) 

O n1.n2.n3 

• F(t - mT)*F(t - n 2 T)*F(n - n 3 T). (15) 

The first term in square brackets (15) is analytic, being the cube of an 
analytic signal. The other two terms in (15) are not analytic, since their 



t The ranges of the indices ni and n2 in (12) and (13) are assumed to be the same. 
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Fourier transforms are not necessarily zero for negative frequencies. The 
analytic signal resulting from passing x(t) 3 through bandpass filter 3 can 
be written by applying Theorem 2 to the first term of (15) and Theorem 
1 to the second and third terms. The resulting analytic signal is the sum 
of three analytic signals, U 3 (t), Uu(t), and Unit), which have the fol- 
lowing forms: 

U 3 (t) m e j«*fct £ A(m)A(nM(n s ) 

n1.n2.n3 

• G s (t - rnT,t - n 2 T,t - n 3 T). (16) 

Unit) - e' Wct £ A( ni )A(n 2 )A(n3)* 

• G u (t - mT,t - n 2 T,t - n 3 T). (17) 

l/i 2 (*)=e-> 2 *^ E A(ni)*A(i» 2 )*A(n 3 ) 

n1.n2.n3 

• G 12 (t - mT,t - n 2 T,t - n 3 T). (18) 

The G( ) signals are complex, and the spectra of the analytic signals 
U 3 it), Unit), and U\ 2 it) are all confined to the range < / < 2/ c by 
bandpass filter 3. 
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